♻️ [Maintenance] Refactor Payments Domain to Use asyncpg Instead of aiopg #8462

pcrespov · 2025-10-06T14:31:31Z

What do these changes do?

This PR migrates database access from aiopg to asyncpg in the following components:

The payments domain within the webserver
The simcore_postgres_database.utils_payments module and its associated test suite

In detail summary (AI generated)

This pull request refactors the payments database utility and its usage across the codebase to fully adopt SQLAlchemy's async API, removing legacy aiopg types and error handling. It also updates test cases and web server modules to use the new async interfaces and corrects model naming for autorecharge logic. These changes modernize the codebase, improve compatibility with SQLAlchemy, and simplify transaction handling.

Database API modernization:

Replaced all usage of aiopg.sa.connection.SAConnection and related types with sqlalchemy.ext.asyncio.AsyncConnection and SQLAlchemy's async result types in utils_payments.py, updating function signatures and result handling accordingly. [1] [2] [3] [4] [5] [6]
Updated error handling in payment transaction functions to catch sqlalchemy.exc.IntegrityError instead of custom aiopg_errors.UniqueViolation.

Test suite updates:

Refactored all payment transaction tests to use AsyncEngine and async context management for database connections, replacing legacy connection types and adapting result assertions to new SQLAlchemy row objects. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]

Web server module updates:

Removed references to aiopg_errors.DatabaseError in exception handling for guest user and project owner removal logic, relying solely on asyncpg and domain-specific errors. [1] [2] [3]

Autorecharge model and API corrections:

Renamed PaymentsAutorechargeDB to PaymentsAutorechargeGetDB and updated all references and API conversion functions to use the new model name, ensuring consistency between DB and API layers. [1] [2] [3] [4] [5] [6]
Updated autorecharge DB module to use async engine and transaction context utilities for database operations.

These changes collectively improve code maintainability, test reliability, and compatibility with modern async database patterns.

Related issue/s

part of migrate code from aiopg to asyncpg #4529

How to test

cd packages/postgres-database
make install-dev
pytest -vv tests/test_models_payments_transactions.py

cd services/web/server
make install-dev
pytest -vv tests/unit/**/wallets/**/test_*.py

Dev-ops

None

codecov · 2025-10-06T14:33:28Z

Codecov Report

❌ Patch coverage is 63.51351% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.97%. Comparing base (9d5b558) to head (156dc6d).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8462      +/-   ##
==========================================
+ Coverage   87.42%   87.97%   +0.54%     
==========================================
  Files        1912     1861      -51     
  Lines       74965    73182    -1783     
  Branches     1333     1063     -270     
==========================================
- Hits        65540    64380    -1160     
+ Misses       9026     8474     -552     
+ Partials      399      328      -71

Flag	Coverage Δ
integrationtests	`64.14% <29.62%> (-0.01%)`	⬇️
unittests	`86.58% <63.51%> (+0.51%)`	⬆️

Components	Coverage Δ
pkg_aws_library	`∅ <ø> (∅)`
pkg_celery_library	`∅ <ø> (∅)`
pkg_dask_task_models_library	`∅ <ø> (∅)`
pkg_models_library	`93.07% <ø> (ø)`
pkg_notifications_library	`85.20% <ø> (ø)`
pkg_postgres_database	`87.94% <90.90%> (-0.02%)`	⬇️
pkg_service_integration	`∅ <ø> (∅)`
pkg_service_library	`70.92% <0.00%> (ø)`
pkg_settings_library	`∅ <ø> (∅)`
pkg_simcore_sdk	`84.95% <ø> (ø)`
agent	`93.53% <ø> (ø)`
api_server	`91.78% <ø> (ø)`
autoscaling	`95.71% <ø> (ø)`
catalog	`92.36% <ø> (ø)`
clusters_keeper	`99.13% <ø> (ø)`
dask_sidecar	`92.35% <ø> (ø)`
datcore_adapter	`97.94% <ø> (ø)`
director	`75.81% <ø> (ø)`
director_v2	`90.93% <ø> (+0.01%)`	⬆️
dynamic_scheduler	`96.71% <ø> (∅)`
dynamic_sidecar	`90.43% <ø> (ø)`
efs_guardian	`89.74% <ø> (ø)`
invitations	`91.41% <ø> (ø)`
payments	`92.71% <50.00%> (ø)`
resource_usage_tracker	`92.20% <ø> (+0.16%)`	⬆️
storage	`86.74% <ø> (+0.29%)`	⬆️
webclient	`∅ <ø> (∅)`
webserver	`87.42% <64.81%> (-0.23%)`	⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d5b558...156dc6d. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mergify · 2025-10-06T15:02:05Z

🧪 CI Insights

Here's what we observed from your CI run for 156dc6d.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on `master`	Retries	🔍 CI Insights	📄 Logs
`CI`	`unit-tests`	Base branch is broken, but retries were needed. Could be early signs of flakiness 👀		`1`	View	View

… consistency in autorecharge API

…to use asyncpg engine

… consistency and update database connection handling to use asyncpg engine

…ne and improve connection handling in tests and payment completion logic

…ency and update references across the codebase

pcrespov · 2025-10-06T16:51:37Z

@mergify queue

Copilot

Pull Request Overview

This PR migrates the payments domain from legacy aiopg to modern asyncpg database access patterns. The refactoring modernizes database interactions by adopting SQLAlchemy's async API throughout the payments system while maintaining backwards compatibility.

Replace aiopg.sa.connection.SAConnection with sqlalchemy.ext.asyncio.AsyncConnection across all payment modules
Update error handling from aiopg_errors to sqlalchemy.exc exceptions
Modernize database result handling using SQLAlchemy's async result methods

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
services/web/server/src/simcore_service_webserver/projects/models.py	Remove aiopg imports and type aliases
services/web/server/src/simcore_service_webserver/payments/_onetime_db.py	Migrate to asyncpg engine and async connection patterns
services/web/server/src/simcore_service_webserver/payments/_onetime_api.py	Update database engine usage and model references
services/web/server/src/simcore_service_webserver/payments/_methods_db.py	Comprehensive migration to async patterns with new connection handling
services/web/server/src/simcore_service_webserver/payments/_methods_api.py	Update model type references
services/web/server/src/simcore_service_webserver/payments/_autorecharge_db.py	Migrate autorecharge functionality to async patterns
services/web/server/src/simcore_service_webserver/payments/_autorecharge_api.py	Update model references for consistency
services/web/server/src/simcore_service_webserver/garbage_collector/_core_utils.py	Remove aiopg error handling
services/web/server/src/simcore_service_webserver/garbage_collector/_core_guests.py	Remove aiopg error handling
services/payments/src/simcore_service_payments/db/auto_recharge_repo.py	Update autorecharge statement class name
packages/service-library/src/servicelib/aiohttp/db_asyncpg_engine.py	Modernize app key usage with proper typing
packages/postgres-database/tests/test_utils_payments_autorecharge.py	Comprehensive test migration to async patterns
packages/postgres-database/tests/test_models_payments_transactions.py	Update test patterns for async database operations
packages/postgres-database/src/simcore_postgres_database/utils_payments_autorecharge.py	Rename class for consistency
packages/postgres-database/src/simcore_postgres_database/utils_payments.py	Core migration from aiopg to asyncpg patterns

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

packages/postgres-database/tests/test_models_payments_transactions.py

mergify · 2025-10-06T16:51:49Z

…ions.py Co-authored-by: Copilot <[email protected]>

matusdrobuliak66

Thanks for this 🥇

wvangeit

thx

sonarqubecloud · 2025-10-06T18:02:11Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

GitHK

thanks

pcrespov added this to the Cheops milestone Oct 6, 2025

pcrespov self-assigned this Oct 6, 2025

pcrespov added a:webserver webserver's codebase. Assigning the area is particularly useful for bugs a:database associated to postgres service and postgres-database package labels Oct 6, 2025

pcrespov force-pushed the mai/payments-to-asyncpg branch from b9a0f03 to 1efb2c3 Compare October 6, 2025 16:25

pcrespov changed the title ~~WIP: Mai/payments to asyncpg~~ ♻️ [Maintenance] Refactor Payments Domain to Use asyncpg Instead of aiopg Oct 6, 2025

pcrespov added the t:maintenance Some planned maintenance work label Oct 6, 2025

pcrespov added 10 commits October 6, 2025 18:50

migrates _methods_api

db536f6

🐛 Fix: rename PaymentsAutorechargeDB to PaymentsAutorechargeGetDB for…

6ef23fb

… consistency in autorecharge API

🐛 Fix: update database connection handling in autorecharge functions …

dbb11f1

…to use asyncpg engine

🐛 Fix: rename PaymentsTransactionsDB to PaymentsTransactionsGetDB for…

a09d405

… consistency and update database connection handling to use asyncpg engine

🐛 Fix: refactor payment transaction state updates to use asyncpg engi…

88bec0d

…ne and improve connection handling in tests and payment completion logic

minor

10af811

fixes pylint

57e08e5

rm unused exceptions and models using aiopg

15cf699

fixes appkey

a098dcf

🐛 Fix: rename AutoRechargeStmts to AutoRechargeStatements for consist…

eccda81

…ency and update references across the codebase

pcrespov force-pushed the mai/payments-to-asyncpg branch from 4eeccc2 to eccda81 Compare October 6, 2025 16:50

pcrespov marked this pull request as ready for review October 6, 2025 16:50

pcrespov requested review from GitHK, matusdrobuliak66 and sanderegg as code owners October 6, 2025 16:50

pcrespov requested review from bisgaard-itis, Copilot, giancarloromeo and wvangeit October 6, 2025 16:51

pcrespov added the 🤖-automerge marks PR as ready to be merged for Mergify label Oct 6, 2025

Copilot AI reviewed Oct 6, 2025

View reviewed changes

packages/postgres-database/tests/test_models_payments_transactions.py Outdated Show resolved Hide resolved

packages/postgres-database/tests/test_models_payments_transactions.py Outdated Show resolved Hide resolved

pcrespov and others added 2 commits October 6, 2025 19:05

Update packages/postgres-database/tests/test_models_payments_transact…

e149e70

…ions.py Co-authored-by: Copilot <[email protected]>

Update packages/postgres-database/tests/test_models_payments_transact…

0208ce7

…ions.py Co-authored-by: Copilot <[email protected]>

pcrespov enabled auto-merge (squash) October 6, 2025 17:05

matusdrobuliak66 approved these changes Oct 6, 2025

View reviewed changes

wvangeit approved these changes Oct 6, 2025

View reviewed changes

Merge branch 'master' into mai/payments-to-asyncpg

156dc6d

giancarloromeo approved these changes Oct 6, 2025

View reviewed changes

GitHK approved these changes Oct 7, 2025

View reviewed changes

pcrespov merged commit 313b982 into ITISFoundation:master Oct 7, 2025
144 of 148 checks passed

pcrespov deleted the mai/payments-to-asyncpg branch October 7, 2025 08:30

This was referenced Oct 28, 2025

🚀 Pre-release master -> staging_Imparable1 #8430

Closed

🚀 Release v1.87.0 #8432

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

♻️ [Maintenance] Refactor Payments Domain to Use asyncpg Instead of aiopg #8462

♻️ [Maintenance] Refactor Payments Domain to Use asyncpg Instead of aiopg #8462

Uh oh!

pcrespov commented Oct 6, 2025 •

edited

Loading

Uh oh!

codecov bot commented Oct 6, 2025 •

edited

Loading

Uh oh!

mergify bot commented Oct 6, 2025 •

edited

Loading

Uh oh!

pcrespov commented Oct 6, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Oct 6, 2025 •

edited

Loading

Uh oh!

matusdrobuliak66 left a comment

Uh oh!

wvangeit left a comment

Uh oh!

sonarqubecloud bot commented Oct 6, 2025

Uh oh!

GitHK left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

♻️ [Maintenance] Refactor Payments Domain to Use asyncpg Instead of aiopg #8462

♻️ [Maintenance] Refactor Payments Domain to Use asyncpg Instead of aiopg #8462

Uh oh!

Conversation

pcrespov commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

In detail summary (AI generated)

Related issue/s

How to test

Dev-ops

Uh oh!

codecov bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mergify bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 CI Insights

✅ Passed Jobs With Interesting Signals

Uh oh!

pcrespov commented Oct 6, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟠 Waiting for conditions to match

Uh oh!

matusdrobuliak66 left a comment

Choose a reason for hiding this comment

Uh oh!

wvangeit left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Oct 6, 2025

Quality Gate passed

Uh oh!

GitHK left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pcrespov commented Oct 6, 2025 •

edited

Loading

codecov bot commented Oct 6, 2025 •

edited

Loading

mergify bot commented Oct 6, 2025 •

edited

Loading

mergify bot commented Oct 6, 2025 •

edited

Loading